WORLD INTELLECTUAL PROPERTY ORGANIZATION 
International Bureau 




per 

INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(51) International Patent Classification 5 : 




(11) International Publication Nnmber: 


WO 93/13916 


B25J 19/02, A61B 1/04 


Al 






B25J3/04 




(43) International Publication Date: 


22 July 1993(22.07.93) 



(21) International Application Nnmber: PCT/US93/0O3O9 

(22) International Filing Date: 14 January 1993 (14.01.93) 



(30) Priority data: 

07/823,932 



21 January 1992 (21.01.92) US 



(71) Applicant: SRI INTERNATIONAL [US/US]; 333 Raven- 

swood Avenue, Menlo Park, CA 94025-3493 (US). 

(72) Inventor: GREEN, Philip, S. ; 585 California Way, Red- 

wood City, CA 94062 (US). 

(74) Agents: LANGE, Richard, P.; SRI International, 333 Rav- 
enswood Avenue, Menlo Park, CA 94025-3493 (US) et 
al. 



(81) Designated States: CA, JP, European patent (AT, BE, CH, 
DE, DK, ES, FR, GB, GR, IE, IT, LU, MC, NL, PT, 
SE). 



Published 

With international search report 
With amended claims. 



(54) Title: TELEOPERATOR SYSTEM AND METHOD WITH TELEPRESENCE 




A teleoperator system with telepresence is shown which includes right and left hand controllers (72R and 72L) for control 
of right and left manipulators (24R and 24L) through use of a servomechanism that includes computer (42). Cameras (46R and 
46L) view workspace (30) from different angles for production of stereoscopic signal outputs at lines (48R and 48L). In response 
to the camera outputs a 3-dimensional top-to-bottom inverted image (301) is produced which, is reflected by mirror (66) toward 
the eyes of operator (18). A virtual image (30V) is produced adjacent control arms (76R and 76L) which is viewed by operator (18) 
looking in the direction of the control arms. By locating the workspace image (30V) adjacent the control arms (76R and 76L) the 
operator is provided with a sense that end effectors (40R and 40L) carried by manipulator arms (34R and 34L) and control arms 
(76R and 76L) are substantially integral. This sense of connection between the control arms (76R and 76L) and end effectors 
(40R and 40L) provide the operator with the sensation of directly controlling the end effectors by hand. By locating visual display 
(246) adjacent control arms (244R and 244L) image (2401) of the workspace is directly viewable by the operator (Figs. 12 and 13). 
Use of the teleoperator system for surgical procedures is also disclosed (Figs. 7-9 and 13). 
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TELEOPERATOR SYSTEM AND METHOD WITH TELEPRESENCE 

FIELD OF THE INVENTION 

This invention relates generally to teleoperator 
method and apparatus, and particularly to those which 
include means for providing the operator of remote 
5 apparatus with the same sense as working directly with 
his hands at the worksite. 

BACKGROUND OF THE INVENTION 

Teleoperating, which is well known, includes the 
human performance of tasks at a remote location using 

10 manipulators. Telepresense includes providing the 
teleoperator with the same feedback and control that he 
would have were he actually at the worksite carrying out 
the operation with his own hands. Telepresence operation 
generally includes use of a stationary visual display, 

15 particularly a stereographic visual display of the remote 
workspace. Stereoscopic television systems are well 
known as shown, for example, in U.S. Patent Numbers 
4,562,463 and 4,583,117 and in U.K. Patent Application 

GB 2,040,134. 

20 Remote manipulators employing stereoscopic TV 

viewing together with force feedback also are well known 
as shown, for example, in an article entitled, 
"Controlling Remote Manipulators Through Kinesthetic 
Coupling," Bejczy et al, Computers in Mechanical 

25 Engineering, July 1983, pps. 48-60, and in an article 
entitled, "Stereo Advantage for a Peg-In-Hole Task Using 
a Force-Feedback Manipulator" by E.H. Spain, SPIE Vol. 
1256 Stereoscopic Displays and Applications, 1990, pps. 
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244-254. In the Bejczy et al article, force-torque 
feedback is disclosed. Also, in U.S. Patent Number 
3,921,445, a manipulator which includes force, torque and 
slip sensors of a type which may be employed with the 
5 present invention is shown. 

Even though the operator of prior art manipulators 
is provided with a stationary three-dimensional image of 
the workspace, and manual controllers for control of the 
manipulators are provided with feedback, the operator is 

10 not provided with a sense of actually being present at 
the worksite. The present invention is directed to a 
viewing arrangement for use in a remote manipulation 
system which substantially adds to the operator's sense 
of presence at the remote manipulator site. 

15 SUMMARY AND OBJECTS OF THE INVENTION 

An object of this invention is the provision of an 
improved teleoperator system and method which include an 
improved viewing system to enhance the operator's sense 
of presence at remote manipulators .controlled by the 

20 operator from a remote location. 

An object of this invention is the provision of 
an improved teleoperator system and method of the above- 
mentioned type wherein an image of manipulator end 
effectors for viewing by the operator are sensed by the 

25 operator as comprising an integral part of hand- 
controllers used by the operator to control the end 
effectors, thereby giving the operator a, strong sense of 
presence at the worksite. 

An object of this invention is the provision of an 

30 improved teleoperator system and method of the above- 
mentioned type which is well adapted for use in a wide 
variety of applications including military, industrial, 
biomedical, and the like. 

The present invention includes manipulators located 

35 at a worksite and which are controlled by hand-operated 
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means at a remote operator control station. End 
effectors at the manipulators are used for manipulating 
objects located in a workspace at the worksite, and 
force-torque feedback is employed for transmitting back 
5 to the operator mechanical resistance encountered by the 
end effectors. stereographic visual display means 
provide the operator with an image of the workspace. In 
accordance with the present invention, the image is 
located adjacent the hand-operated means so that the 

10 operator looks in the direction of the hand-operated 
means for viewing the image adjacent the hand-operated 
means. Either a real or virtual image of the workspace 
may be provided adjacent the hand-operated means. 
Display means for display of a real image may be located 

15 adjacent the hand-operated means for direct viewing of 
the real image by the operator. For display of a virtual 
image of the workspace, a mirror is located between the 
operator's eyes and the hand-operated means. In this 
case, display means provide a real image which is 

20 inverted from top to bottom, which inverted image is 
viewed via the mirror, which mirror inverts the image and 
provides the operator with a virtual image of the 
workspace, which appears to be located adjacent the hand- 
operated means. By locating the image of the workspace 

25 adjacent the hand-operated means the operator is provided 
with a sense that the end effectors and hand-operated 
means are substantially integral despite, the fact the end 
effectors are located at the worksite and the hand- 
operated means are located at the remote operator's 

30 station. A stereophonic sound system may be included to 
provide the operator with stereophonic sound from the 
worksite. Video camera means are provided for viewing 
the workspace from which an image of the workspace is 
obtained. Various other sensors and associated 

35 responders may be located at the worksite and operator's 
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station, respectively, for transmission of pressure, 
tactile, heat, vibration and similar information for 
enhanced telepresence operation. 

Depending upon the application, different scaling 
5 may be provided in the transmission of information 
between the operator's station and worksite. For 
example, for microassembly , microsurgery and like 
operations involving small part manipulation, optical 
and/or video magnification may be employed to provide an 
10 enlarged 3 -dimensional image for viewing by the operator. 
With similar scaling between the hand operated means and 
manipulators, the perception of the operator is 
substantially that which a miniature operator would have 
were he at the worksite. 

15 B-RTEF DESrRTPTIOW OF TWK DRAWINGS 

The invention, together with other objects and 
advantages thereof, will be better understood from the 
following description considered with the accompanying 
drawings. It will be understood that the drawings are 
20 for purposes of illustration and example only, and that 
the invention is not limited thereto. In the drawings, 
wherein like reference characters refer to the same parts 
in the several views, 

Fig. 1 is a diagrammatic showing of a teleoperator 
25 system embodying the present invention including side 
elevational views of a worksite and remote control 
operator's station; 

Fig. 2 is an enlarged rear elevational view of the 
operator's station taken substantially along line 2-2 of 

3d Figure l; 

Fig. 3 is an enlarged rear elevational view of the 
worksite taken substantially along line 3-3 of Figure 1; 

Fig. 4 is a simplified side elevational view which 
is similar to Figure 1 and showing dimensional 
35 relationships between elements at the worksite and 
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elements at the operator's station; 

Fig. 5 is a diagrammatic view to illustrate visual 
perception by a miniature virtual eye, and Fig. 6 is a 
diagrammatic view to illustrate visual perception by the 
5 operator when image magnification is employed; 

Fig. 7 is a diagrammatic view which is similar to 
that of Fig. 1 but showing the teleoperator system used 
for telepresence surgery; 

Fig. 8 is a rear elevational view of the operator's 
10 station shown in Fig. 7; 

Fig. 9 is a rear elevational view of the worksite 
shown in Fig. 7; 

Figs. 10 and 11 are fragmentary side elevational 
views of modified forms of operator's station and 
15 manipulator, respectively, having increased degrees of 
freedom; 

Fig. 12 is a side elevational view of a modified 
form of operator's station wherein display means are 
positioned for direct viewing by the operator; 
20 Fig. 13 is a rear elevational view of the modified 

form of operator's station shown in Fig. 12; and 

Fig. 14 shows a fragmentary portion of the insertion 
portion of an endoscope for use with the present 
invention. 

25 Reference now is made to Figs. 1-3 wherein the 

teleoperator system is shown to include an operator's 
station 20 (Figs. 1 and 2) and worksite / 22 (Figs. 1 and 
3). An operator 18 at the operator's station controls 
manipulator means 24 at the remote worksite. Manipulator 

30 means 24, comprising right and left manipulators 24R and 
24L, respectively, are used for manipulating objects, 
such as object 26 which is shown located on a platform, 
or base, 28 within a workspace 30 shown in broken lines. 
For purposes of illustration only, and not by way of 

35 limitation, the right manipulator 24R is shown to 
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comprise a housing 32R affixed to base 28 and from which 
housing a telescopic arm 34R extends. The inner end 34R1 
of arm 34R is mounted for pivotal movement in any pivotal 
direction using conventional mounting means. For 
5 example, the inner end of arm 34R may be mounted for 
pivotal movement about a horizontal pivot axis 36 which 
pivot axis, in turn, is adapted for pivotal movement 
about vertical axis 38. 

Arm 34R includes telescopic inner section 34R1 and 

10 outer section 34R2, which outer section is adapted both 
for axial movement into and out of inner section 34R1 and 
for rotation about its longitudinal axis. An end 
effector 40R is carried at the outer end of the arm 
which, for purposes of illustration, is shown to comprise 

15 a gripper. Motor means, not shown, control pivotal 
movement of arm 34R about pivot axes 36 and 38, axial and 
rotary movement of outer arm section 34R2 along and about 
the longitudinal axis of the arm, and opening and closing 
of gripper 4 OR. The motor means, together with motor 

20 control circuits for control of the motors, may be 
included in housing 32R. The motors are under control of 
a computer 42 connected thereto through right manipulator 
interface 44R and the above-mentioned motor control 
circuits. 

25 The left manipulator 24L is of substantially the 

same design as the right manipulator 24R and the same 
reference numerals, but with the suffix. L instead of R, 
are used to identify similar parts. For purposes of 
illustration, the left end effector 40L, shown in Fig. 3, 

30 is seen to comprise cutting blades which operate to cut 
in the manner of a pair of scissor blades. 

The worksite is provided with a pair of video 
cameras 46R and 46L for viewing workspace 30 from 
different angles for production of stereoscopic signal 

35 outputs therefrom at lines 48R and 48L. The angle y 
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between the optical axes of the cameras shown in Fig. 3 
is substantially equal to the operator's interocular 
viewing angle y of an image of the workspace as shown in 
Fig. 2. 

5 The video camera outputs at lines 48R and 48L are 

supplied to an image memory 50 for momentary storage of 
video fields of right and left images from the cameras. 
Fields of right and left images from image memory 50 are 
alternately supplied through left/right switch means 52 

10 to visual display means 54, such as a television monitor, 
for alternate display of the two images at the face 54A 
of the monitor. Timing and control means 56 provide 
timing and control signals to various elements of the 
system, including elements included in the stereographic 

15 display system, for signal timing and control of the 
system. If digital storage means 50 are employed, then, 
conversion of the camera signal outputs to digital signal 
form by analog to digital converter means prior to 
storage, and conversion of the digital signal output from 

20 left/right switch means to analog signal form in 
preparation for display at monitor 54 may be employed. 

An electrooptical device 58 at the face of the 
display means 54 controls polarization of light received 
from display means 54 under control of a left/right 

25 synchronizing signal from timing and control unit 56. 
The left and right image fields are viewed by operator 18 
wearing a pair of passive polarized glasses 60 having 
right and left polarizing elements 62 and 64 polarized in 
orthogonal directions. The polarization of light from 

30 display 54 through electrooptical device 58 is 
synchronized field by field such that the right field is 
occluded from the left eye and the left field is occluded 
from the right eye for stereographic viewing by the 
operator. Other means for stereographic viewing of left 

35 and right image fields are well known, including, for 



WO 93/13916 



PCI7US93/00309 



example, those using active stereographic glasses, which 
may be used in the practice of this invention to provide 
the operator with a stereoscopic view of the remote 
workspace. 

5 The vertical deflection coil connections for monitor 

54 are reversed, causing the monitor to scan from bottom 
to top thereby creating a top-to-bottom inverted image 
301 of workspace 30. Letters a, b, c and d are used to 
identify corresponding corners of the workspace 30 and 

10 inverted workspace image 301. The inverted workspace 
image 301 is viewed by the operator via a mirror 66 at 
the top of a table 68, which mirror inverts image 301 to 
return the image as viewed by the operator to an upright 
position. Looking downwardly in the direction of the 

15 mirror, the operator views a virtual image 30V of 
workspace 30. In accordance with one aspect of the 
present invention, the image viewed by the operator, 
which in the Fig. 1-3 embodiment comprises a virtual 
image, is located adjacent controller means 70 used by 

20 the operator for control of manipulator means 24 at the 
worksite. 

Controller means 70 are shown located beneath the 
table top 68 and include right and left controllers 72R 
and 72L for control of the respective right and left 

25 manipulators 24R and 24L. The right and left controllers 
are of substantially the same design so that a 
description of one applies to both. , As with the 
manipulators, the suffixes R and L are used to 
distinguish elements of the right controller from those 

30 of the left controller. For purposes of illustration, 
and not by way of limitation, the right controller 72R is 
shown to comprise a housing 74R affixed to the bottom of 
table top 68 and from which hand-operated means 76R in 
the form of a telescopic control arm, or stick, extends. 

35 The right and left control arms 76R and 76L are 
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provided with the sane degrees of freedom as the 
associated manipulator arms 34R and 34L, respectively. 
For example, the inner end of control arm 76R is mounted 
for pivotal movement about a horizontal pivot axis, 
5 corresponding to manipulator pivot axis 36, which axis, 
in turn, is adapted for pivotal movement about an 
intersecting vertical axis, corresponding to manipulator 
axis 38. Control arm 76R also includes inner section 
76R1 and outer section 76R2, which outer section is 

10 adapted both for axial movement into and out of inner 
section 76R1 and for rotation about its longitudinal 
axis- It will be apparent that the control arm 76R is 
provided with the same four degrees of freedom as the 
associated manipulator arm 34R. Additionally, sensor 

15 means 78R are located adjacent the outer end of outer arm 
section 76R2 for use in controlling gripping action of 
gripper 40R. Similar sensor means 78L adjacent the outer 
end of control arm'76L are adapted for use in controlling 
operation of scissor blades 40L. 

20 Right and left controllers 72R and 72L are included 

in a servomechanism system wherein mechanical motion of 
control arms 76R and 76L controls the position of 
manipulator arms 34R and 34L, and pressure on sensor 
means 78R and 78L controls opening and closing of end 

25 effectors. 40R and 40L, respectively. In Fig. 1, right 
and left hand controller interfaces 80R and 80L, 
respectively, are shown for connection of, the controllers 
to computer 42. Servomechanisms for control of 
mechanical motion at a remote location are well known, 

30 including those which provide force and torque feedback 
from the manipulator to the hand-operated controller 
means. Any suitable prior art servomechanism may be used 
in the practice of the present invention, with those 
incorporating force and torque feedback being 

35 particularly preferred for telepresence operation of the 
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system. in the illustrated system, right and left 
microphones are included at the worksite, outputs from 
which microphones are amplified by right and left 
amplifiers and supplied to right and left speakers at the 
5 operators' station for providing a stereophonic sound 
output to provide the operator with an audio perspective 
present at the workspace. in Fig. 1, only the right 
channel of the stereophonic system is shown including 
right microphone 82R, right amplifier 86R and right 
10 speaker 88R. The left microphone and speaker are located 
directly behind the respective right microphone and 
speaker at the worksite and operator's control station as 
viewed in Fig. l. Obviously, earphones may be provided 
for use by the operator in place of the speakers which 
15 would help to block out external noises at the operator's 
control station. Also, in Fig. l a light shield 54B at 
the monitor is shown for blocking direct viewing of the 
monitor face by the operator. 

Reference now is made to Fig. 4 wherein a simplified 
diagrammatic view of the system illustrated in Figs. 1-3 
is shown and wherein various lengths and angular 
positions are identified by reference characters. in 
Fig. 4, the optical path length between the cameras and 
a point F at the workspace is identified by reference 
25 character L. A corresponding path length between the 
operator's eyes and point F at the virtual image of the 
workspace is identified by the distance a+b, where a is 
the distance from the eyes of the operator to mirror 66, 
and b is the distance from the mirror to point F at the 
30 virtual image, other dimensions shown include the height 
G of the cameras above the pivot point of manipulator arm 
34R and corresponding height g of the operator's eyes 
above the pivot point of control arm 76R. with the 
control arm 76R at length d, the manipulator arm 34R 
adjusts to length D. Similarly, with the control arm 76R 



20 



35 
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at an angle p A with the vertical, the manipulator arm 34R 
is positioned at the same angle from vertical. The angle 
from vertical at which the cameras view the workspace and 
the eyes view the virtual image of the workspace is 
5 identified by a. 

Between elements of the worksite and operator 
station, the following relationships pertain: 



where k is a scale factor constant. 
When k equals 1 such that a + b = L, d = D and g = G, no 
scaling of worksite dimensions is required. 

Any scale factor may be employed, the invention not 
15 being limited to full-scale manipulation. For example, 
the worksite can be small, including microscopic in size, 
in which case the optical parameters, including distance 
to object, interocular distance and focal length, and 
mechanical and dimensional parameters are appropriately 
20 scaled. 

By using appropriate scaling and image magnification 
and force and torque feedback, and by locating the image 
30V of the workspace 30 adjacent hand-operated control 
means 76R and 76L, the operator is provided with a strong 

25 sense of directly controlling the end effectors 4 OR and 
40L. The operator is provided with a sense that the end 
effectors 4 OR and 40L and respective control arms 76R and 
76L are substantially integral. This same sense of 
togetherness of the hand-operated control means and end 

30 effectors is not provided in prior art arrangements 
wherein the image viewed by the operator is not located 
adjacent the hand-operated control means. Even where the 
prior art includes stereoscopic viewing and force and 
torque feedback, there is a feeling of disconnectedness 

35 of the hand motions from the visual image object being 



10 



a + b =kL, 
d = kD 
g = kG 



and 



(1) 
(2) 
(3) 
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worked upon. The present invention overcomes this sense 
of disconnectedness by locating the workspace image where 
the operator's hands appear to exercise direct control 
over the end effectors. 
5 For small-scale manipulation, such as required for 

surgical applications, it is desired to replicate the 
visual experience that a miniature observer would have 
were he closely adjacent the actual worksite. In Fig. 5, 
the virtual eye 90 of a hypothetical miniature observer 

10 is shown viewing an actual workspace. Light from a 
source at a point X, Y, Z in the actual workspace 
produces a stimulus on the miniature observer's eye 90 at 
a point identified as X'/M. In Fig. 6, an eye 92 of an 
actual operator is shown viewing an enlarged image of the 

15 virtual workspace produced by means of a video camera 94 
used to view the actual workspace. The illustrated 
camera includes a light-receiving lens 96 and solid state 
imaging device such as a charge-coupled-device (CCD) 
array 98 where the point light source at X, Y, Z is shown 

20 imaged at point X L , Y L , Z L . With correct scaling, a 
corresponding light source is produced at point MX if MY if 
MZ^ at either the real or apparent position of the face 
of the visual display which, due to stereoscopic 
operation of the system appears to the operator to 

25 originate from point MX, MY, MZ corresponding to point 
X,Y,Z at the actual workspace. At the retina of the 
actual eye 92, a stimulus is produced / at point X' at 
proportionately the same position as point X'/M at eye 90 
of the hypothetical observer. This relationship is 

30 ensured by selecting a correctly scaled camera distance 
and lens focal length such that the optical magnification 
Mq = M/My. where M is the desired overall magnification 
and My is the video magnification. A typical video 
magnification, My, which equals the ratio of the CCD- 

35 array 98 width to the display width, is about 40. 
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Reference now is made to Figs. 7 through 9 wherein 
a modified form of this invention is shown for medical 
use. Here, right and left manipulators 100R and 100L are 
shown which are under control of right and left 
5 controllers 102R and 102L, respectively. Elements of the 
imaging system are substantially the same as those 
employed in the imaging system illustrated in Figs. 1-3 
described above except that an enlarged virtual image 
104V of actual workspace 104 is provided for viewing by 

10 the operator. Also, servomechanism elements for 
connection of the right and left controllers 102R and 
102L to the respective manipulators 100R and 100L are 
substantially the same as those described above with 
reference to Figs. 1-3. In the illustrated arrangement, 

15 the right and left manipulators are of substantially the 
same construction as are the right and left controllers, 
such that a description of one manipulator and one 
controller applies to both. Again, suffixes R and L are 
used to distinguish between right and left elements 

20 thereof. 

The manipulators include outer control sections 
100RA and 100LA and insertion sections 10 ORB and 100LB, 
-which insertion sections are adapted for insertion into 
a body cavity through cylindrical tubes, or cannulas, not 

25 shown. For purposes of illustration, the manipulators 
are shown inserted through the abdomen wall 106 of a 
subject. As is well understood, fpr laparoscopic 
surgical procedures, wall 106 is separated from internal 
organs by insufflation wherein a gas is introduced into 

30 the abdomen by any suitable means not shown. Manipulator 
motors and associated motor control circuits are 
contained in the outer control sections 100RA and 100LA 
of the manipulators for control of the insertion section. 
The manipulators, together with a laparoscope 108 for 

35 viewing organs within the cavity, are carried by a fixed 
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rail 110 forming part of a surgical table upon which the 
subject is supported. 

The insertion sections 10 ORB and 100LB of the 
manipulators may be of substantially the same design as 
5 manipulator arms 34R and 34L described above with 
reference to the Figs. 1-3 embodiment. The insertion 
sections are of relatively small size for use inside the 
body. Insertion section 100RB includes telescopic inner 
section 112R1 and outer section 112R2, which outer 

10 section is adapted for both axial movement into and out 
of inner section 112R1 and for rotation about its 
longitudinal axis. End effectors 114R and 114L are 
carried at the outer ends of the respective right and 
left sections 112R2 and 112L2 for manipulation of organ 

15 116. The inner section 112R1 is adapted for pivotal 
movement about intersecting perpendicular axes located 
substantially at point P where the insertion section 
intersects wall 106. Exclusive of operation of end 
effectors 114R and 114L the manipulator arms each are 

20 provided with four degrees of freedom, the same as in the 
embodiment shown in Figs 1-3. End effectors 114R and 
114L simply may comprise, essentially, microsurgical 
instruments with their handles removed including, for 
example, retractors, electrosurgical cutters and 

25 coagulators, microforceps, microneedle holders, 
dissecting scissors, blades, irrigators, and sutures. 

Laparoscope 108 for viewing the workspace 104 is 
shown comprising an outer operating section 108A and 
insertion section 108B. The outer end section 120 of 

30 insertion section 108B is axially and rotatably movable 
within the inner end 122 thereof, and is provided with a 
pair of image transmission windows 124,124 for 
stereoscopic viewing of workspace 104. The laparoscope 
also is provided with illuminating means, not shown for 

35 illuminating the workspace, and with liquid inlet and 
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outlet means, not shown, for flow of liquid past the 
windows. Video camera means within section 108A are 
responsive to light received through the viewing windows 
for generation of left and right electronic images at 
5 output lines 48R and 48L for connection to image memory 
50. A magnified 3 -dimensional image 1041 is produced at 
display means 54 for viewing by the operator wearing 
cross-polarized glasses 60 via mirror 66. As with the 
embodiment shown in Figs. 1-3, a virtual image 104V of 

10 the workspace 104 is produced adjacent control arms 130R 
and 130L of controllers 102R and 102L. Control arms 130R 
and 130L are of the same type as control arms 76R and 76L 
included in the Figs. 1-3 embodiment described above. 
They include telescopic inner and outer sections 132R1 

15 and 132R, and 132L1 and 132L2. Sensor means 134R and 
13 4L located adjacent the outer ends of the control arms 
control operation of end effectors 114R and 114L, 
respectively, in the manner described above with 
reference to Figs. 1-3. It here will be noted that the 

20 angle from vertical at which the image is viewed by the 
operator need not equal the angle from vertical at which 
the object is viewed by the cameras. In the arrangement 
illustrated in Figs. 7-9, the operator is shown to view 
the image 104V at an angle e from vertical (Fig. 7) 

25 whereas the object 116 is shown as viewed directly 
downwardly. With no external reference, the sense of 
vertical within a body is not particularly great, and no 
confusion is produced in the mind of the operator as a 
result of the different observer and camera viewing 

30 angles relative to vertical. 

With the Figs. 7-9 embodiment, not only is a 
magnified virtual image 104V of the workspace provided 
for viewing by the operator, but control arms 13 OR and 
130L of greater length than the length of the manipulator 



35 
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insertion sections 100KB and 100LB axe employed. 
Servomechanism scaling of axial movement of the 
telescopic control arms is provided such that axial 
extension or retraction thereof results in a smaller 
5 extension or retraction of the telescopic insertion 
sections. Angular pivotal motion of the control arms 
13 OR and 13 0L produces the same angular pivotal motion of 
insertion sections 100RB and 100LB, and rotational 
movement of the end sections 132R2 and 132L2 of the 
10 control arms produces the same rotational motion of end 
sections 112R2 and 112L2 of the insertion sections of the 
right and left manipulators, without scaling. This 
embodiment of the invention, with its magnified image, is 
of particular use in the area of microsurgery, and 
15 especially in those cases where the surgeon cannot reach 
an area by hand because of size constraints. 

The present invention is not limited to use with 
manipulators having any particular number of degrees of 
freedom. Manipulators with different degrees of freedom 
20 which are well known in the art may be used in the 
practice of this invention. In Figs. 10 and 11, to which 
reference now is made a controller 140 and manipulator 
142, respectively, are shown which include a wrist joint 
to provide the same with additional freedom of movement. 
25 The illustrated controller 140 includes a housing 144 
affixed to the bottom of table top 68 upon which table 
mirror 66 is. located. An enlarged virtual image 146V of 
actual workspace 146 is provided adjacent the operator's 
hand 148 viewable by the operator when looking downwardly 
30 onto the mirror 66 in a manner described above. 

A control arm 150L comprising inner and outer 
sections 150L1 and 150L2, respectively, is mounted within 
housing 144 for pivotal movement in any pivotal direction 
as indicated by intersecting double-headed arrows 152 and 
35 154. The outer section 150L2 is adapted for axial 
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movement into and out of inner section 150L1 in the 
direction of double-headed arrow 156. It also is adapted 
for rotation about its longitudinal axis in the direction 
of double-headed arrow 158. in this embodiment, the 
5 control arm includes an end section 160 pivotally 
attached to outer section 150L2 by wrist joint 162 for 
pivotal movement in the direction of double-headed arrow 
164. End section 160 comprises axially aligned inner and 
outer sections 160A and 160B, the outer section 160B of 
10 which is rotatable about its longitudinal axis in the 
direction of double-headed arrow 166. As with the above- 
described arrangements, sensor means 168 are located 
adjacent the free end of the control arm for operation of 
an end effector 170 at manipulator 142 shown in Fig. ll. 
15 Referring to Fig. 11, end effector 170 is shown to 

comprise a pair of movable jaws attached to a wrist 172 
comprising axially aligned links 172A and 172B. Outer 
link 172B is rotatable about its. longitudinal axis 
relative to inner link 172A by motor means, not shown, in 
20 the direction of double-headed arrow 166M in response to 
rotation of section 160B of the hand-operated control 
unit in the direction of arrow 166. Wrist link 172A is 
pivotally attached to manipulator forearm 174 for pivotal 
movement in the direction of double-headed arrow 164M in 
25 response to pivotal movement of end section 160 of the 
hand-operated control means about pivot axis 162 . 
Forearm 174 is longitudinally axially ' movable in the 
direction of double-headed arrow 156M in response to 
axial movement of outer section 150L2 of control arm 150L 
30 in the direction of double-headed arrow 156. It also is 
rotatable about its longitudinal axis in the direction of 
double-headed arrow 158M in response to rotation of outer 
section 150L2 of control arm 150L in the direction of 
double-headed arrow 158. Additionally, it is pivotally 
35 movable about point 176 in the directions of double- 
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headed arrows 152M and 154M in response to pivotal 
movement of control arm 150L in the directions of double- 
headed arrows 152 and 154, respectively. For biomedical 
use, such as remote laparoscopic surgery, pivot point 176 
5 is substantially located at the level of abdominal wall 
178 through which the manipulator extends. In Fig. 11, 
manipulator arm 174 is shown extending through a cannula 
180 which penetrates the abdominal wall. 

The outer operating end of the manipulator is 

10 adapted for attachment to a supporting rail, not shown, 
of the surgery table upon which the subject is supported. 
It includes an end effector drive motor 182 for opening 
and closing of gripper 170. Wrist drive motor 184 
controls pivotal movement of wrist 172 in the direction 

15 of double-headed arrow 164M, and extension drive motor 
186 controls axial movement of manipulator arm 174 in the 
direction of double-headed arrow 15 6M. Forearm pivotal 
control motors and linkages, identified generally by 
reference numeral 188, provide for pivotal movement of 

20 arm 174 about pivot point 176 in the directions of arrows 
152M and 154M. Pivotal motion about point 176 is 
provided by simultaneous lateral movement of the outer 
operating end of the manipulator and pivotal movement of 
arm 174. Movements are coordinated such that the center 

25 of rotation of forearm 174 is fixed in space at point 176 
at the level of the abdominal wall. 

Controller 140 and manipulator 142, are included in 
a system such as shown in Figs. 7, 8 and 9 which includes 
a second controller and manipulator for use by the 

30 operator's right hand, and associated servomechanism 
means of any suitable type, not shown, for remote control 
of the manipulators by the hand-operated controllers. 
Video camera means at the worksite, such as shown in Fig. 
9, together with display means, such as shown in Fig. 7, 

35 are employed for providing the operator with an image of 
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the workspace at a location adjacent the left and right 
hand-operated control means. By using manipulators with 
a wrist joint, an added degree of freedom is provided for 
increased maneuverability and usefulness thereof. 
5 However, as noted above, the present invention is not 
limited to use with manipulators with any particular 
degree of freedom. 

Reference now is made to Figs. 12 and 13 wherein a 
modified form of this invention is shown which provides 

10 for direct viewing of a 3 -dimensional image 2401 of a 
workspace, not shown. In Figs. 12 and 13, only the 
operator's station is shown, which includes right and 
left controllers 242R and 242L and associated right and 
left hand-operated means 244R and 244L which may be of 

15 the same type as controllers and control arms described 
above. The operator's station is adapted for remote 
control of manipulators which also may be of the above- 
described type. The 3 -dimensional image 2401 of the 
workspace is provided by visual display means 246 in 

20 conjunction with electrooptical device 58 at the face of 
the display means and cross-polarized glasses 60 worn by 
the operator, to which display means left and right video 
fields from left and right video cameras that view the 
workspace are alternately supplied, all in the manner 

25 described in detail above. End effector and object 
images 248 and 250, respectively, are shown within the 
workspace image as viewed by video .cameras at the 
worksite. The display means 246 is located adjacent the 
left and right hand-operated means 244R and 244L for 

30 direct viewing by the operator. With this arrangement, 
the end effector and object images together with the 
hand-operated means 244R and 244L are simultaneously 
viewable by the operator. Since the hand-operated means 
also are visible, the operator is provided with a visual 

35 sense of connection between the end effector means and 
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hand-operated means whereby they appear substantially as 
being integral. 

Reference now is made to Fig. 14 wherein the distal 
end portion, or tip, 260 of the insertion section of an 
5 endoscope is shown which is of substantially the same 
type as shown in the above-mentioned publication entitled 
"Introduction to a New Project for National Research and 
Development Program (Large-Scale Project) in FY 1991" 
which endoscope may be used in the practice of the 

10 present invention. The insertion end of the endoscope 
includes a pair of spaced viewing windows 262R and 262L 
and an illumination source 264 for viewing and 
illuminating a workspace to be observed. Light received 
at the windows is focused by objective lens means, not 

15 shown, and transmitted through fiber-optic bundles to a 
pair of cameras at the operating end of the endoscope, 
not shown. The camera outputs are converted to a 3- 
dimensional image of the workspace which image is located 
adjacent hand-operated means at the operator's station, 

20 not shown. Right and left steerable catheters 268R and 
268L pass through accessory channels in the endoscope 
body, which catheters are adapted for extension from the 
distal end portion, as illustrated. End effectors 27 OR 
and 270L are provided at the ends of the catheters which 

25 may comprise conventional endoscopic instruments. Force 
sensors, not shown, also are inserted through the 
endoscope channels. Steerable catheters which include 
control wires for controlling bending of the catheters 
and operation of an end effector suitable for use with 

30 this invention are well known. Control motors for 
operation of the control wires are provided at the 
operating end of the endoscope, which motors are included 
in a servomechanism of a type described above for 
operation of the steerable catheters and associated end 

35 effectors from a remote operator's station. As with the 
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other embodiments, the interfacing computer in the 
servomechanism system remaps the operator's hand motion 
into the coordinate system of the end effectors, and 
images of the end effectors are viewable adjacent the 
5 hand-operated controllers in a manner described above. 
With this embodiment, the operator has the sensation of 
reaching through the endoscope to put his hands directly 
on the end effectors for control thereof. Endoscopes of 
different types may be employed in this embodiment of the 

10 invention so long as they include one or more accessory 
channels for use in control of end effector means, and 
suitable viewing means for use in providing a visual 
display of the workspace. For example, gastric, 
colonscopic, and like type, endoscopes may be employed. 

15 The invention having been described in detail in 

accordance with requirements of the Patent Statutes, 
various other changes and modifications will suggest 
themselves to those skilled in this art. For example, as 
noted above, the invention may include the use of tactile 

20 feedback to provide the subtle sensations for palpation 
and for manipulating tissues and instruments. To provide 
this feedback, tactile sensor arrays may be included on 
the end effectors which are coupled to tactile sensor 
stimulator arrays on the hand-operated control means, 

25 which reproduce the tactile sensation on the operator's 
hands. A variety of transduction technologies for 
teleoperator tactile sensing are known including 
resistive/conductive, semiconductor, piezoelectric 
capacitive and photoelectric. Hand-operated control 

30 means and manipulators of different types may be employed 
using a wide variety of well-known mechanisms and 
electromechanical elements including, for example, 
gimbals, linkages, pulleys, cables, drive belts and 
bands, gears, optical or electromagnetic position 

35 encoders, and angular and linear motors. Force feedback 
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to the operator requires use of body contact with, hand- 
operated control means. Both hand grip type hand 
controllers such as those illustrated, and control brace 
type hand controllers are well adapted for use with the 
5 present invention for force feedback to the operator. 
Control brace hand controllers include use of structures 
with positive sensors mounted on the operator at joints 
for measuring joint angles. Force feedback then can be 
applied to each joint. Similarly, light fabric gloves 
10 with variable-resistance or fiber-optic flex sensors 
mounted on the joints for measuring bending of individual 
fingers may be used. Gloves of this type also may be 
provided with force feedback to provide for telepresence 
interaction with real objects. Regardless of the type of 
15 hand-operated control means employed, an image of the 
workspace is produced adjacent thereto to provide the 
operator with a sense that the end effector means and 
hand-operated control means are substantially integral. 
Also, as noted above, servomechanisms of many different 
20 types are well known in the robotic and teleoperator 
system arts, and the invention is not limited to any 
particular type. Those that include force and torque 
feedback to the operator are preferred to contribute to 
a telepresence sense of operation. In addition, many 
25 different means for producing a stereoscopic image of the 
workspace are known. For example, instead of using two 
cameras, a single camera may be employed together with 
switched cross-polarizing elements in the image receiving 
path. In this case, a pair of spaced stereoscopic lenses 
30. are used for viewing the workspace from different angles 
and providing first and second images thereof to the 
camera. In the Fig. 9 arrangement, wherein a laparoscope 
is shown, other types of endoscopes may be used for 
viewing the workspace. As noted above, the invention is 
35 not limited to any particular application or use. In the 
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biomedical field, uses include, for example, open 
surgery, including surgery from a remote location, 
microsurgery, and minimum invasive surgery such as 
laparoscopic and endoscopic surgery. Laboratory use 
5 including microscopic manipulation also is contemplated. 
Industrial use of the invention include, for example, 
hazardous materials handling, remote operations, 
microassembly, and the like. Military and undersea use 
of the teleoperator system of this system are apparent. 
10 It is intended that the above and other such changes and 
modifications shall fall within the spirit and scope of 
the invention defined in the appended claims. 
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I CLAIM: 

1. A teleoperator system for manipulating objects 
located in a workspace at a worksite by an operator from 
a remote control operator's station comprising, 
5 manipulator means at the work site including end 

effector means for manipulation of an object at the 
workspace, 

controller means including hand-operated means at 
the control station connected to said manipulator means 
10 for remote control of the manipulator means by control of 
the hand-operated means by the operator, 

video camera means for viewing the workspace, and 
means responsive to the video camera means for 
producing an image of the workspace which image is 
15 located adjacent the hand-operated means for viewing by 
the operator in the direction of said hand-operated means 
during operation of the hand-operated means and providing 
the operator with a sense that the end effector means and 
hand-operated means are substantially integral. 

20 2. A teleoperator system as defined in Claim 1 wherein 
said video camera means comprises means for viewing the 
workspace from different angles for production of 
stereoscopic signal outputs, 

said means for producing an image comprising means 

25 for producing a 3-dimensional image of the workspace in 
response to said stereoscopic signal outputs. 

3. A teleoperator system as defined in Claim 2 wherein 
the angle between the optical axes of said means for 
viewing the workspace is substantially equal to the 

30 operator's interocular viewing angle of the image. 

4. A teleoperator system as defined in Claim 2 wherein 
said hand-operated means comprises a control arm mounted 
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for movement in any pivotal direction, the image of the 
end effector means and position of the movable control 
arm as sensed by the operator providing the operator with 
a sense that the end effective means comprises an 
5 extension of the control arm. 

5. A teleoperator system as defined in Claim 4 wherein 
said manipulator includes an arm mounted for movement in 
any pivotal direction, angular movement of the control 
arm by the operator producing substantially the same 

10 angular movement of the manipulator arm. 

6. A teleoperator system as defined in Claim 5 wherein 
the length of path from the eyes of the operator to the 
image of the workspace and length of control arm are 
related by a factor k to the optical path length from the 

15 video camera means to the workspace and length of 
manipulator arm, respectively, where k is a constant. 

7. A teieoperatpr system as. defined in Claim 2 wherein 
said image viewable by the operator comprises a virtual 
image. 

20 8. A teleoperator system as defined in Claim 7 wherein 
said means for producing an image includes display means 
for producing a top-to-bottom reversed 3-dimensional, 
real image of the workspace, 

a mirror positioned in front of the operator so as 

25 to receive the reversed real image from the display means 
and to reflect the same toward the operator's eyes for 
viewing of said virtual image in the mirror. 

9. A teleoperator system as defined in Claim 8 wherein 
the display means is located above the mirror and faces 
30 toward the mirror. 
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10. A teleoperator system as defined in Claim 7 wherein 
the display means faces downwardly at an angle from 
vertical . 

11. A teleoperator system as defined in Claim 1 wherein 
5 said manipulator means comprises first and second 

manipulators and said controller means comprises first 
and second hand-operated means for remote control of said 
first and second manipulators, respectively. 

12. A teleoperator system as defined in Claim 1 
10 including an endoscope comprising an operating section 

and an insertion section insertable in a cavity in which 
the workspace is located and having observing window 
means through which the workspace is viewed by the video 
camera means. 

15 13. A teleoperator system as defined in Claim 12 wherein 
said manipulator means includes an operating section and 
insertion section insertable in said cavity for 
manipulation of an object in the cavity. 

14. A teleoperator system as defined in Claim 13 
20 wherein said manipulator means comprise first and second 
manipulators and said controller means comprise first 
and second hand-operated means for remote control of said 
first and second manipulators, respectively. 

15. A teleoperator system as defined in Claim 14 
25 including means for attaching the operating sections of 
the manipulators and endoscope to a support rail of a 
surgical table for support thereof during surgery. 

16. A teleoperator system as defined in Claim 1 
including force feedback from the manipulator means to 
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the hand-operated means which, together with said image 
of the workspace, provides for visual and force 
telepresence operation of the system. 

17. A teleoperator system as defined in Claim 1 
5 including first and second stereophonic microphones for 

picking up sounds from the workspace, and 

first and second speakers at the remote control 
station responsive to outputs from said respective first 
and second microphones for producing a stereophonic sound 
10 output and providing the operator with an audio 
perspective present at the workspace. 

18. A teleoperator system as defined in Claim 1 wherein, 
said means for producing an image of the workspace 
includes display means for producing a real image of the 

15 workspace adjacent the hand operated means for direct 
viewing by the operator. 

19. a teleoperator system as defined in Claim 1 
including an endoscope having an accessory channel, and 
wherein 

20 said manipulator means comprising a steerable 

catheter extending through the channel, said end effector 
means being carried by said steerable catheter, said 
workspace being viewed by said video camera means through 
said endoscope. 

25 20. In a method of performing surgery by an operator at 
a remote location which includes, 

inserting the insertion end of an endoscope and of 
first and second manipulators into a subject, 

supporting the operating ends of the endoscope and 
30 first and second manipulators, 

employing said endoscope for viewing an object 
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inside the subject by video camera means, 

controlling said first and second manipulators by 
respective first and second hand-operated controllers 
remotely located from said manipulators, and 
5 presenting an image of the object viewed by the 

video camera means at a location adjacent the first and 
second hand-operated controllers for viewing by the 
operator during operation of the first and second hand- 
operated controllers and providing the operator with a 
10- sense that the first and second manipulators, and 
respective first and second hand-operated controllers are 
substantially integral. 

21. In a method as defined in Claim 20 wherein a 
stereoscopic video camera is employed for viewing the 

15 object by said endoscope, and 

wherein the image presented is a 3 -dimensional 

image. 

22. In a method as defined in Claim 20 which includes 
presenting a magnified image of the object for viewing by 

20 the operator. 

23. In a method as defined in Claim 20 wherein the image 
viewed by the operator comprises a virtual image. 

24. In a method as defined in Claim 23 wherein the image 
viewed by the operator comprises a real image. 

25 25. In a method as defined in Claim 20 wherein the 
endoscope and first and second manipulators are inserted 
into the subject's abdomen, said method including 
insufflating the abdomen. 
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AMENDED CLAIMS 

[received by the International Bureau on 8 June 1993 (08.06.93); 
original claims 1-8,10,11,13-21,24 and 25 amended; 
new claims 26-32 added; other claims unchanged (7 pages)] 

1. A teleoperator system for manipulating objects located 
in a workspace at a worksite by an operator from an 
operator's control station located remotely from the 
worksite comprising, 
5 manipulator means at the worksite including a first 

movable manipulator arm and first end effector means at 
one end of said first manipulator arm for manipulation of 
an object at the workspace, 

controller means including first hand-operated 
10 movable control means at the operator's control station 
connected by servomechanism means to said manipulator 
means for remote control of said first movable 
manipulator arm by control of said first hand-operated 
movable control means by the operator, said controller 
15 means including stationary support, means at the 
operator's control station for supporting said first 
hand-operated movable control means, 

video camera means for viewing the workspace, and 
means responsive to the video camera means for 
20 producing an image of the workspace which said workspace 
image is located adjacent said first hand-operated 
movable control means for viewing by the operator in the 
direction of said first hand-operated movable control 
means during operation of said first hand-operated 
25 movable control means, proprioceptive awareness by the 
operator of the operator's hand position in combination 
with said workspace image providing the operator with a 
sense that the first end effector means and first hand- 
operated movable control means are substantially 
30 integral, said video camera means being located relative 
to said first movable manipulator arm such that the 
direction of movement of said one end of said first 
movable manipulator arm relative to said video camera 
means in response to operation of said first hand- 
35 operated movable control means is substantially the same 
as the direction of movement of said first hand-operated 
movable control means relative to the operator's eyes. 
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2. A teleoperator system as defined in Claim 1 wherein 
said video camera means comprises means for generating 
stereoscopic signal outputs, 

said image producing means comprising means for 
5 producing a 3 -dimensional image of the workspace in 
response to said stereoscopic signal outputs. 

3. A teleoperator system as defined in Claim 2 wherein 
said video camera means comprises means for viewing the 
workspace from different angles such that the angle 

10 between the optical axes of said workspace viewing means 
is substantially equal to the operator's interocular 
viewing angle of the image. 

4. A teleoperator system as defined in Claim 1 wherein 
said first hand-operated movable control means comprises 

15 a first hand-operated movable control arm, which first 
hand-operated movable control arm and first movable 
manipulator arm are mounted for movement in any pivotal 
direction, the image of the first end effector means and 
position of the first movable control arm as sensed by 

20 the operator providing the operator with a sense that the 
first end effector means comprises an extension of said 
first hand-operated movable control arm. 

5. A teleoperator system as defined in Claim 4 wherein 
angular movement of said first hand-operated movable 

25 control arm by the operator produces substantially the 
same angular movement of said first movable manipulator 
arm. 

6. A teleoperator system as defined in Claim 5 wherein 
path length from eyes of the operator to said workspace 

30 image and length of said first hand-operated movable 

control arm are related by a scale factor k to the 
optical path length from the video camera means to the 
workspace and length of said first movable manipulator 
arm, respectively, where scale factor k corresponds to 
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magnification of said workspace image. 

7. A teleoperator system as defined in Claim 1 wherein 
said workspace image viewable by the operator comprises 
a virtual image. 

5 8. A teleoperator system as defined in Claim 7 wherein 
said means for producing an image includes display means 
for producing a top-to-bottom reversed 3-dimensional, 
real image of the workspace, 

a mirror positioned in front of the operator so as 
10 to receive the reversed real image from the display means 
and to reflect the reversed real image toward the 
operator's eyes for viewing of said virtual image in the 
mirror. 

9. A teleoperator system as defined in Claim 8 wherein 
15 the display means is located above the mirror and faces 
toward the mirror. 



10. A teleoperator system as defined in Claim 8 wherein 
the display means faces downwardly at an angle from 
vertical. 

11. A teleoperator system as defined in Claim 1 wherein 
said manipulator means further comprises a second 
manipulator arm and second end effector means at one end 
of said second manipulator arm and said controller means 
further comprises a second hand-operated movable control 
means for remote control of said second manipulator arm. 



12. A teleoperator system as defined in Claim 1 
including an endoscope comprising an operating section 
and an insertion section insertable in a cavity in which 
the workspace is located and having observing window 
means through which the workspace is viewed by the video 
camera means. 
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13. A teleoperator system as defined in Claim 12 wherein 
at least a portion of said first manipulator arm is 
insertable in said cavity for manipulation of an object 
in the cavity. 

5 14. A teleoperator system as defined in Claim 13 wherein 
said manipulator means further comprises a second movable 
manipulator arm and second end effector means at one end 
of said second manipulator arm and said controller means 
further comprises second hand-operated movable control 
10 means for remote control of said second manipulator arm, 
at least a portion of said second movable manipulator arm 
being insertable into said cavity for manipulation of an 
object in the cavity. 

15. A teleoperator system as defined in Claim 14 
15 including means for attaching said first and second 

manipulator arms and said endoscope to a surgical table 
for support thereof during surgery. 

16 . A teleoperator system as defined in Claim 1 
including force feedback from said manipulator means to 

20 said first hand-operated movable control means which 
provides for force telepresence operation of the system. 

17. A teleoperator system as defined in Claim 1 
including first and second stereophonic microphones for 
picking up sounds from said workspace, and 

25 first and second speakers at the remote control 

station responsive to outputs from said first and second 
microphones, respectively, for producing a stereophonic 
sound output and providing the operator with an audio 
perspective present at the workspace. 

30 18. A teleoperator system as defined in Claim 1 wherein 

said means for producing an image of the workspace 
includes display means for producing a real image of the 
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workspace adjacent the first hand-operated movable 
control means for direct viewing by the operator. 

19. A teleoperator system as defined in claim 1 
including an endoscope having at least one accessory 

5 channel, and wherein 

said manipulator means further comprises at least 
one steerable catheter extending through said accessory 
channel, said end effector means being carried by said 
steerable catheter, said workspace being viewable by said 
10 video camera means through said endoscope. 

20. In a method of performing surgery by an operator at 
a remote location which includes, 

inserting the insertion end of first and second 
manipulators comprising first and second movable arms 
15 with first and second end effectors, respectively, into 
a subject, 

inserting the insertion end of an endoscope into the 
subject, 

supporting the operating end of the endoscope and 
20 said first and second manipulators, 

employing said endoscope for viewing an object 
inside the subject by video camera means, 

controlling said first and second movable arms of 
said manipulators by respective first and second hand- 
25 operated control arms of controllers remotely located 
from said first and second manipulators, 

controlling said first and second end effectors by 
respective first and second means at said first and 
second control arms, respectively, and 
30 presenting an image of the object viewed by the 

video camera means at a location adjacent the first and 
second hand-operated control arms for viewing by the 
operator during operation of the first and second hand- 
operated control arms and for providing the operator with 
J 5 a sense that the first and second end effectors and 
respective first and second hand-operated control arms 
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are substantially integral. 

21. In a method as defined in Claim 20 which includes 
employing a stereoscopic video camera for viewing the 
object by said endoscope, and 

5 wherein the image presented is a 3 -dimensional 

image. 

22. In a method as defined in Claim 20 which includes 
presenting a magnified image of the object for viewing by 
the operator. 

10 23. In a method as defined in Claim 20 wherein the image 
viewed by the operator comprises a virtual image. 

24. In a method as defined in Claim 20 wherein the image 
viewed by the operator comprises a real image. 

25. In a method as defined in Claim 20 wherein the 
15 endoscope and said first and second manipulators are 

inserted into the subject's abdomen, including the step 
of insufflating the subject's abdomen. 

26. A teleoperator system as defined in Claim 1 wherein 
said controller means includes means at said first hand- 

20 operated movable control means connected by 

servomechanism means to said first end effector means for 
remote operation of said first end effector means by 
operator control of said means at said first hand- 
operated movable control means. 

25 27. A teleoperator system as defined in Claim 26 wherein 
said first hand-operated movable control means comprises 
a first hand-operated movable control arm to which said 
means for remote operation of said first end effector 
means is affixed. 

30 28. A teleoperator system as defined in Claim 1 wherein 
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said workspace image comprises a magnified image of said 
workspace . 

29. A teleoperator system as defined in Claim 28 wherein 
movement of said first hand-operated movable control 

5 means is related to resultant movement of said first 
movable manipulator arm by a scale factor corresponding 
to magnification of said workspace image. 

30. A teleoperator system as defined in Claim 1 wherein 
said first movable manipulator arm and first hand- 

10 operated movable control means have the same degrees of 
freedom. 

31. A teleoperator system as defined in Claim 1 
including a microphone for picking up sounds from said 
workspace , and 

15 a speaker at the operator's control station 

responsive to output from said microphone. 

32. A teleoperator system as defined in Claim 8 wherein 
said display means comprises a monitor with a scan that 
is inverted relative to the workspace image from the 

20 video camera means. 
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